The present invention relates generally to systems for optically determining the direction of an object relative to an imaging system or for optically determining the position of an object such as a pointer and, more particularly, to the use of imaging systems placed about a computer display device to monitor the motions of a pointer.
There are so-called "touch screen" systems wherein a user interacts with a computer system by touching or pointing to various locations within a touch screen area, typically associated with a display such as a CRT, or other display device. The touch screen serves as easily used data input apparatus because the operator can quickly and easily feed data into a computer by indicating various specific positions of the touch screen area.
As employed herein, the term "pointer" refers to any suitable pointing object, externally illuminated or self-illuminated which is easily moved relative to a two-dimensional plane surface or area. By way of example and not limitation, the pointer may comprise a pen, stylus, finger, or any typically long and slim element.
As employed herein, the term "touch screen" means apparatus used to locate the position of a pointer within a generally planar viewing field. In accordance with the invention, such a viewing field may be an area defined on or near the surface of solid material, such as near an inert panel, or may be a geometric plane defined in space or in the air.
Such "touch screens" have many possible applications, for example: pointing to or selecting a particular image or region on a video screen or on a panel, plate or tablet to select or indicate a specific item, informational element, letter, number, symbol, parameter line, region or characteristic; to locate or track a pointer held generally perpendicular to the plane and moved along the plane for plotting data, drawing a graph or picture or for laying out a network or diagram; for indicating or tracking the relative motion or coordinates of movement or trajectory of a moving object; for use as a custom keyboard in an electronic input module; and so forth.
For convenience, the terminology "Stylus Detection" is employed herein to refer to the subject invention; it will be appreciated, however, that "stylus" refers to any "pointer" or other object whose direction relative to an imaging system or position is to be determined. In addition, the term "camera" is employed herein as a synonym to imaging system, where a "camera" may be characterized as an electronic device for capturing either one- or two-dimensional images and converting them to electrical signals for further processing.
In overview, the Stylus Detection system of the invention employs two or more cameras placed strategically about a computer display to obtain the position of the computer operator's hand or an object such as a pen or stylus. The precise position of the hand or object is determined using trigonometric triangulation. Pointer position, as well as signals derived from the change of position or the rate of change of position, can be interpreted as commands to the computer system.
The Stylus Detection devices of the invention most closely resemble what collectively have been called touch screens in the prior art. Touch screens themselves have been largely divided into two groups: overlay, in which a device sized to fit the display is placed over or attached to the display screen itself; and non-overlay, in which no such display-screen-covering device is required. Although not touch screens, the Stylus Detection devices of the invention could be classed as non-overlay.
At the present time, by far the most commonly used touch screen technologies are overlay. Overlay touch screens usually require placing a specially prepared device over the computer display and using the device to determine the position, in two dimensions, of the operator's finger or some other stylus. Typical examples involve one- or two-layer arrangements of glass or plastic sheets which have been coated or patterned with special materials so that the position of a stylus or finger touch can be determined by making resistance or capacitance measurements on the device. Another approach broadcasts acoustic waves on the surface of a specially prepared piece of glass and determines two-dimensional positions by timing when the waves have been attenuated by a touch from a finger or other sufficiently compliant stylus. This latter device lays claim to a third-dimension sensing capability, but the "third dimension" actually relates to firmness of touch, since no detection at all takes place until the finger or stylus is firmly in contact with the display screen.
In addition to being unable to detect the position of a pointing device before it is actually in contact with the device surface, overlay technologies suffer from several other problems. Any nominally transparent overlay material reduces light transmission from the display and also is an increased source of distracting reflections from the face of the display. Reflections are greatly increased for technologies such as resistive and capacitive touch screens since the required overlays are coated with a metallic film that acts as a mirror. These overlays are often covered with diffusive coatings used to reduce the added reflections, but the diffusion also reduces the sharpness of the display. Thus the net result of applying overlay technology is a dimmer display which may also have a fuzzy appearance or disturbing surface reflections. These problems are especially serious when dealing with liquid crystal displays, which are characteristically dimmer than desired even without the overlay.
Overlays also increase the problem of parallax. The thickness of the overlay increases the distance between the display image plane and the point of touch. When the user moves his head, and thus the position of his eyes, a fixed point being touched on the surface of the touch screen appears to move relative to the display image plane underneath. As a perpendicular distance between the image plane and the point of touch is increased, this parallax effect is magnified accordingly.
Overlay sensors also present a major logistical problem for manufacturers. Typically, each model of computer display requires a specially designed and configured overlay sensor. Even displays of the same nominal size tend to differ slightly in actual visible display area, in curvature of the display surface, or even in the available space behind the display bezel, making the required overlay designs differ. This means that a large inventory and an on-demand design capability be maintained unless the touch screen manufacturer wishes to severely limit the display models to which his technology can be applied. Furthermore, retrofitting displays for overlay touch screens is usually a complicated and specialized job, not easily performed in the field.
Non-overlay touch screens have thus far been largely unsuccessful in the marketplace, due largely to the awkwardness of the approaches which have heretofore been taken. One such device includes a linear array of perhaps twenty-five infrared light-emitting diodes placed along one side of a display and a matched linear array of infrared detectors along the other side. When touching the display screen, the operator's finger breaks a light beam, thus indicating position on the vertical axis of the display. A similar arrangement at the top and bottom of the display is employed to determine horizontal axis position. As implemented, this prior art system provides relatively low precision in determining two-dimensional position. Because the diodes have to be placed a significant distance out from the display screen, parallax is also a problem. While this system obviously has the ability to detect the approach of a finger or stylus some distance above the display, it actually is forced to use this above-display point as precisely equivalent to touching the display. This detection of a touch before it actually happens results in a poor feel and lack of tactile response, and contributes greatly to the lack of success of the sensor.
Light pens are one of the oldest display interaction technologies available. Light pens are not usually classed with touch screens, mainly for historical reasons. They require a special stylus (the light pen) which typically must be tethered to a controller via a cord carrying an electronic or optical signal from the display unit. Light pens require that a delicate balance be struck between the persistence of the display and the achievable horizontal resolution, a problem which is aggravated by color displays, which require three phosphors, typically having different persistences. Light pens are completely non-functional in situations in which there are significant dark (unlighted) areas in the target display, especially problematical with monochrome displays. These impediments have prevented the wide-spread use of light pens despite their early availability.
It will be appreciated that, in systems of the type employing imaging systems placed strategically about a computer display, each imaging system includes a photodetector, such as a charge-coupled device (CCD) having a plurality of elements organized as an array, and a lens to focus an image of the pointer object onto the photodetector. Assuming the position of the object image on the sensor array can be accurately determined, the direction of the object relative to the imaging system can be determined in a straightforward manner since position within the photodetector array is related to the angular position within the field of view of the imaging system, typically in a nearly proportional manner. Given directions relative to at least two imaging systems, pointer object position can be determined by triangulation, as is disclosed for example in Denlinger U.S. Pat. No. 4,782,328.
Prior to the present invention, the resolution of such systems has generally been limited by the number of photodetector elements, although interpolation in some cases has been used to infer an image position between two detector elements, for example. As described in greater detail hereinbelow, such is a highly inefficient use of a CCD array.